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DESCRIPTION 

IMAGE PROCESSING METHOD AND IMAGE PROCESSING APPARATUS 

TECHNICAL FIELD 

The present Invention relates to a technique of 
presenting a virtual object superimposed on a physical 
space to an observer. 

BACKGROUND ART 

Some conventional 3D CAD browsing tools display a 
CAD object as a virtual object on a display such as a 
CRT or on the display unit of an HMD in combination 
with the VR technology. 

In recent years, studies about mixed reality (to 
be referred to as "MR" hereinafter} which aims at 
seamless combination of the physical space and virtual 
space have extensively been done. MR aims at 
coexistence of the physical space and the virtual 
reality (to be referred to as VR hereinafter) world 
which can conventionally be experienced only in a 
situation isolated from the physical space and provides 
MR by combining an image in the physical space with an 
image in the virtual space (e.g., Japanese Patent 
Laid-Open No. 11-136706). 

In the former case, since the sense of actual 
size is absent, it is difficult to grasp the size of a 
designed object. In addition, since a pointing device 
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sucli as a mouse is used to observe a target CAD object 
from various directions, complex and time-consuming 
operations are necessary to change the position and 
orientation of the object and the viewpoint and line of 
5 sight of the observer. 

In the latter case as well, since comparison to 
an object which exists in the physical world is 
impossible, the size and the like of a designed object 
are hard to grasp. 

10 Generally, to browse 3D CAD data, many operations 

are required to observe an object from various 
directions, select some parts, or set the non-display 
mode. In the latter system, however, it is difficult 
to mount a complex user interface because the program 

15 is based on cumbersome 3D CG. 

DISCLOSURE OF INVENTION 
The present invention has been made in 
consideration of the above -described problems, and has 
20 as its object to provide a technique for more easily 

browsing and manipulating a virtual object in a virtual 
space. 

In order to achieve an object of the present 
invention, for example, an image processing method of 
25 the present invention comprises the following 
arrangement . 

That is, an image processing method of presenting 
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a virtual object superimposed on a physical space to an 
observer, characterized by comprising: 

a holding step of holding, in a memory , 
information of a state of each virtual object included 
in a virtual space; 

an association step of associating at least one 
virtual object included in the virtual space with at 
least one position/orientation sensor which is held in 
a hand of the observer and manipulated; 

a layout step of laying out, in the virtual 
space, the virtual object associated with the 
position/orientation sensor in the association step in 
accordance with a position and orientation of the 
position/orientation sensor itself, which are 
determined on the basis of a measurement result by the 
position/orientation sensor; 

a presentation step of presenting the state of 
each virtual object included in the virtual space to 
the observer on the basis of the information held in 
the holding. step ; 

an input step of inputting an operation to change 
a desired one of the states presented in the 
presentation step; and 

an updating step of updating the information in 
accordance with the operation input in the input step. 

In order to achieve an object of the present 
invention, for example, an image processing method of 
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the present invention comprises the following 
arrangement . 

That is, an image processing method including 

acquiring position and orientation information of 
an observer, 

acquiring an image of a physical space; 

generating an image of a virtual space in 
accordance with the position and orientation 
information of the observer, and 

compositing the image of the physical space and 
the image of the virtual space and displaying the 
composited image on a head mounted display worn by a 
user, characterized by comprising: 

generating an operation panel image and 
compositing the operation panel image with the image of 
the physical space and the image of the virtual space; 

acquiring position information of an operation 
unit operated by the observer; and 

updating the operation panel image in accordance 
with a positional relationship between the operation 
panel image and the operation unit, 

wherein in the operation panel image, a part 
selected by the operation unit is enlarged. 

In order to achieve an object of the present 
invention, for example, an image processing apparatus 
of the present invention comprises the following 
arrangement . 
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That is , an image processing apparatus for 
presenting a virtual object superimposed on a physical 
space to an observer, characterized by comprising: 

holding unit adapted to hold information of a 
5 state of each virtual object included in a virtual 
space ; 

association unit adapted to associate at least 

one virtual object included in the virtual space with 

at least one position/orientation sensor manipulated by 
10 the observer; 

layout unit adapted to lay out, in the virtual 

space, the virtual object associated with the 

position/orientation sensor by the association unit in 

accordance with a position and orientation of the 
15 position/orientation sensor itself , which are 

determined on the basis of a measurement result by the 

position/orientation sensor; 

presentation unit adapted to present the state of 

each virtual object included in the virtual space to 
20 the observer on the basis of the information held by 

the holding unit; 

input unit adapted to input an operation to 

change a desired one of the states presented by the 

presentation unit; and 
25 updating unit adapted to update the information 

in accordance with the operation input by the input 

unit . 
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Other features and advantages of the present 
invention will be apparent from the following 
description taken in conjunction with the accompanying 
drawings, in which like reference characters designate 
5 the same or similar parts throughout the figures 
thereof. 

BRIEF DESCRIPTION OF DRAWINGS 
The accompanying drawings, which are incorporated 
10 in and constitute a part of the specification, 

illustrate embodiments of the invention and, together 
with the description, serve to explain the principles 
of the invention . 

Fig. 1 is a view showing the outer appearance of 
15 a system according to the first embodiment of the 

present invention, which provides, to an observer, a 
mixed reality space obtained by superimposing a virtual 
space on the physical space and makes it possible to 
browse and manipulate a virtual object in the virtual 
20. space; 

Fig. 2 is a view showing the detailed structure 
of an HMD 100; 

Fig. 3 is a block diagram showing the functional 
arrangement of a computer 400; 
25 Fig. 4 is a flowchart showing processing executed 

by the units of the computer 400; 

Fig. 5 is a view showing a display example of an 
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operation panel window; and 

Fig. 6 is a block diagram showing the basic 
arrangement of a computer 400 according to the second 
embodiment of the present invention. 

5 

BEST MODE FOR CARRYING OUT THE INVENTION 
Preferred embodiments of the present invention 
will now be described in detail in accordance with the 
accompanying drawings . 
10 [First Embodiment] 

Fig. 1 is a view showing the outer appearance of 
a system according to the first embodiment of the 
present invention, which provides, to an observer, a 
mixed reality space obtained by superimposing a virtual 
15 space on the physical space and makes it possible to 
browse and manipulate a virtual object in the virtual 
space. 

Referring to Fig. 1, a transmitter 200 generates 
a magnetic field. A head mounted display (to be 

20 referred to as an HMD hereinafter) 100 is mounted on 

the head of an observer to provide an image obtained by 
compositing the physical space and virtual space in 
front of his/her eyes. The HMD 100 includes cameras 
102a and 102b, display devices 101a and 101b , and a 

25 magnetic receiver 201. 

The cameras 102a and 102b continuously sense the 
physical space viewed from the positions of the right 
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and left eyes of the observer who wears the HMD 100 on 
the head. Each sensed frame image is output to a 
computer 400 of the subsequent stage. 

The display devices 101a and 101b are mounted in 
5 the HMD 100 such that they are located just in front of 
the right and left eyes of the observer when he/she 
wears the HMD 100 on the head. The display devices 
101a and 101b display images based on image signals 
output from the computer 400 of the subsequent stage. 
10 Hence, images generated by the computer 400 are 

provided in front of the right and left eyes of the 
observer. 

The magnetic receiver 201 detects a change in 
magnetic field generated by the transmitter 200 and 

15 outputs a detection result signal to a 

position/orientation measuring device 205 of the 
subsequent stage. The detection result signal 
indicates a change in magnetic field, which is detected 
in accordance with the position and orientation of the 

20 magnetic receiver 201 in a coordinate system (to be 

referred to as a sensor coordinate system hereinafter) 
. in which the origin is set at the position of the 
transmitter 200 , and three axes crossing each other 
perpendicularly at the origin are set as X- , Y- , and 

25 Z-axes. The position/orientation measuring device 205 
obtains the position and orientation of the magnetic 
receiver 201 in the sensor coordinate system on the 
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basls of the signal . The data representing the 
obtained position and orientation is output to the 
computer 400 of the subsequent stage . 

Fig. 2 is a view showing the detailed structure 
5 of the HMD 100, 

An image display device 101 is formed from a 
small liquid crystal display device having a size of 
0.5 to several inches. A free surface prism 103 serves 
as a lens to enlarge the image on the image display 

10 device 101. With this structure, the image displayed 
on the image display device 101 is presented to the 
observer as, e.g., an image having a size of 90 inches 
at a position 2 m ahead. 

An image input device 102 is formed from an image 

15 sensing device such as a CCD camera or CMOS camera. An 
image sensing system prism 104 serves as a lens to 
converge light in the physical space to the image input 
device 102. The image sensing system prism 104 is 
arranged outside the free surface prism 103 so that 

20 their optical axes match. With this structure, the 
disparity between the image input by the image input 
device 102 and the image displayed on the image display 
device 101 can be eliminated, and the image in the 
physical space can be played back without any sense of 

25 discomfort. 

Referring back to Fig. 1, a 3D pointing device 
300 (position/orientation sensor) incorporates a 
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magnetic receiver (not shown). When the observer holds 
the 3D pointing device 300 in hand and changes its 
position and orientation, the magnetic receiver (not 
shown) outputs, to the position/orientation measuring 
5 device 205, a signal (i.e., signal indicating the 

position and orientation of the magnetic receiver (not 
shown) itself in the sensor coordinate system) 
indicating a change in magnetic field, which is 
detected in accordance with the position and 

10 orientation of 3D pointing device 300 in the coordinate 
system, like the magnetic receiver 201. On the basis 
of this signal, the position/orientation measuring 
device 205 can obtain the position and orientation of 
the magnetic receiver in the sensor coordinate system. 

15 The data representing the obtained position and 
orientation is output to the computer 400 of the 
subsequent stage. The observer takes the 3D pointing 
device 300 in hand and manipulates it to change the 
position and orientation of a virtual object (i.e., 

20 virtual object as an observation target) which is 

associated with the 3D pointing device 300 in advance. 
Use of the 3D pointing device 300 will be described 
later. 

A 3D pointing device 302 is used for a purpose 
25 different from that of the 3D pointing device 300 and 
incorporates a magnetic receiver 203, like the 3D 
pointing device 300. Hence, the position and 
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orientation of the 3D pointing device 302 itself in the 
sensor coordinate system can be obtained, like the 3D 
pointing device 300, The observer holds the 3D 
pointing device 302 in hand and manipulates it to, 
5 e.g., designate a section of a virtual object or select 
a part of the virtual object. Use of the 3D pointing 
device 302 will be described later. 

An operation panel display device 304 functions 
as a display device to display the image of an 
10 operation panel {to be described later). The image of . 
the operation panel is generated by the computer 400 of 
the subsequent stage and output. 

An input device 303 is used by the observer to 
input an instruction in accordance with contents 
15 displayed on the operation panel display device 304. 
Use of the input device 303 will be described later. 

The computer 400 executes processing of 
generating image signals to be output to the display 
devices 101a and 101b of the HMD 100 or operation panel 
20 display device 304 or receiving and managing data from 
the position/orientation measuring device 205. 

Fig. 3 is a block diagram showing the functional 
arrangement of the computer 4.00. In this embodiment, 
the respective units shown in Fig- 4 are formed by 
2 5 hardware . 

Image capture units 401R and 401L capture images 
input from the cameras 102a and 102b, respectively. 
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A position/orientation information input unit 404 
receives data output from the position/orientation 
measuring device 205. This data contains data 
representing the position and orientation of the 
5 magnetic receiver 201 in the sensor coordinate system, 
data representing the position and orientation, in the 
sensor coordinate system, of the magnetic receiver 
included in the 3D pointing device 300, and data 
representing the position and orientation, in the 

10 sensor coordinate system, of the magnetic receiver 203 
included in the 3D pointing device 302. 

A 3D CAD data DB (database) 408 stores data (3D 
CG rendering data) to generate the image of a virtual 
object which forms a virtual space. 3D CG rendering 

15 data contains data representing the geographical shape 
and color, texture data, and data representing the 
position and orientation of the virtual object. 

A 3D CG scene graph DB 406 stores scene graph 
data in which the 3D CG rendering data is 

20 hierarchically managed on the basis of the parent-child 
relationship between virtual objects . This DB also 
stores information representing the state of each 
virtual object, including the name and blinking state, 
selected/unselected state, and section display state of 

25 each virtual object. These information can be updated 
by an operation to be described later. The 3D CG scene 
graph DB 406 also manages the positions and 
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orientations (to be referred to as a position and 
orientation of viewpoint hereinafter) of the cameras 
102a and 102b. 

When data representing the position and 
5 orientation, in the sensor coordinate system, of the 
magnetic receiver included in the 3D pointing device 
300 is input from the position/ orientation information 
input unit 404 r a position/orientation calculation unit 
405 obtains the position and orientation, in the 

10 virtual space, of the magnetic receiver included in the 
3D pointing device 300 by known calculation by using 
the received data and conversion data (data 
representing the relationship between the position and 
orientation in the virtual space and those in the 

15 sensor coordinate system; the position and orientation 
in one coordinate system can be converted into those in 
the other coordinate system by the conversion data) 
between the sensor coordinate system and the virtual 
space. The position/orientation calculation unit 405 

20 updates the position and orientation of a virtual 

object associated with the 3D pointing device 300 in 
advance to the obtained position and orientation. This 
updating is done by updating the position and 
orientation of the virtual object in the scene graph 

25 data. 

A virtual object can be associated with the 3D 
pointing device 300 by, e.g., the following method. 
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First, a control node is provided in the scene graph. 
The control node is used to change the position and 
orientation of a virtual object linked to the control 
node on the basis of an externally given value. 
5 In this embodiment, a control node to receive 

data representing the position and orientation, in the 
sensor coordinate system, of the magnetic receiver 
included in the 3D pointing device 300 is provided in 
the scene graph. The node of the virtual object to be 

10 associated with the 3D pointing device 300 is linked to 
the control node. 

When data representing the position and 
orientation, in the sensor coordinate system, of the 
magnetic receiver included in the 3D pointing device 

15 300 is given to the position/orientation calculation 

unit 405. the position/orientation calculation unit 405 
gives this data to the control node. The CD rendering 
unit 407 gives this data to the node of the virtual 
object linked to the control node so that the position 

20 and orientation of the virtual object associated with 
the 3D pointing device 300 are updated to the position 
and orientation of the 3D pointing device 300. 

With this operation, the virtual object can be 
associated with the 3D pointing device 300. The 

25 control node is a known technique, and a detailed 
description thereof will be omitted. 

The number of virtual objects associated with the 
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3D pointing device 300 is not particularly limited. 

When data representing the position and 
orientation, in the sensor coordinate system, of the 
magnetic receiver included in the 3D pointing device 
302 is input from the position/orientation information 
input unit 404, the position/orientation calculation 
unit 405 obtains the position and orientation, in the 
virtual space, of the magnetic receiver included in the 
3D pointing device 302 by known calculation by using 
the received data and conversion data. The 
position/orientation calculation unit 405 updates the 
position and orientation of a virtual object associated 
with the 3D pointing device 302 in advance to the 
obtained position and orientation. This updating is 
done by updating the position and orientation of the 
virtual object in the scene graph data. 

When data representing the position and 
orientation of the magnetic receiver 201 in the sensor 
coordinate system is input from the 

position/orientation information input unit 404, the 
position/orientation calculation unit 405 obtains the 
position and orientation of the magnetic receiver 201 
in the virtual space by known calculation by using the 
received data and conversion data. When data 
representing the position and orientation relationship 
between the magnetic receiver 201 and the cameras 102a 
and 102b is used, the positions and orientations of the 
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cameras 102a and 102b ±n the virtual space can be 
obtained by Jcnown calculation. In this embodiment, the 
data to obtain the positions and orientation of the 
cameras 102a and 102b in the virtual space from the 
position and orientation of the image input device 102 
is given in advance as known data. The thus obtained 
data of the positions and orientation of the cameras 
102a and 102b is managed in the 3D CG scene graph DB 
406. 

A CG rendering unit 407 generates the image of 
the virtual object seen in accordance with the 
positions and orientations of the cameras 102a and 102b 
calculated by the position/orientation calculation unit 
405 (positions and orientations of the cameras 102a and 
102b managed in the 3D CG scene graph DB 406). The 
position and orientation of each virtual object in the 
virtual space are based on data in the 3D CG scene 
graph DB 406. The image of each virtual object is also 
based on data in the 3D CG scene graph DB 406, Hence, 
the image of the virtual object associated with the 3D 
pointing devices 300 and 302 is laid out in the virtual 
space on the basis of the positions and orientations of 
the 3D pointing devices 300 and 30 2. The image is 
viewed in accordance with the positions and 
orientations of the cameras 102a and 102b calculated by 
the position/orientation calculation unit 405- 

Processing of generating the image of a virtual 
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object viewed from a viewpoint having predetermined 
position and orientation is a known technique, and a 
detailed description thereof will be omitted. 
Image compositing units 402R and 402L 
5 respectively superimpose the image of the virtual 
object viewed in accordance with the position and 
orientation of the camera 102a and the image of the 
virtual object viewed in accordance with the position 
and orientation of the camera 102b , which are generated 

10 by the CG rendering unit 407 , on the image of the 

physical space input from the image capture units 401R 
and 401L and output the images to image generation 
units 403R and 403L. With this processing, the image 
of the mixed reality space viewed in accordance with 

15 the position and orientation of the camera 102a and the 
image of the mixed reality space viewed in accordance 
with the position and orientation of the camera 102b 
can be generated. 

The image generation units 403R and 403L convert 

20 the images of the mixed reality space output from the 
image compositing units 402R and 402L into analog data 
and output the data to the display devices 101a and 
101b as image signals, respectively. The images of the 
mixed reality space corresponding to the positions of 

25 the right and left eyes of the observer who wears the 
HMD 100 on the head are displayed in front of the 
respective eyes . 
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An operation information processing unit 409 
generates a panel window by using various kinds of the 
information registered in the 3D CG scene graph DB 406. 
The operation information processing unit 409 also 
5 reflects, on the panel window, a result obtained when 
the operator inputs an instruction on the panel window 
by using the input device 303 or manipulates the 3D 
pointing device 302 and outputs the data of the 
operation panel window after reflection to an operation 

10 panel generation unit 410. When information about the 
virtual object is changed by using the input device 
303, the operation information processing unit 409 
executes a change corresponding to the operation for 
corresponding data in the 3D CG scene graph DB 406. 

15 More specifically, the image of the operation 

panel is displayed on the display screen of the 
operation panel display device 304, as shown in Fig. 5. 
Fig. 5 is a view showing a display example of the 
operation panel window. 

20 Referring to Fig. 5, an operation panel image 500 

indicates, e.g., the assembly tree structure of CAD 
data to select an arbitrary component or enlarge/reduce 
a component. The operation panel. image 500 functions 
as GUI. 

25 Referring to Fig. 5, a region 506 hierarchically 

displays information (component name in Fig. 5) about 
each component of one virtual object in accordance with 
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data in the 3D CG scene graph DB 406. As indicated by 
501 in Fig. 5, one component is selected. The 
component name is displayed in a larger size, as 
indicated by 502, so that the observer can easily 
5 recognize it. 

As indicated by a tab 505 in Fig. 5, information 
about each component is displayed to set 
display /non- display (whether to display the component 
on the display devices 101a and 101b of the HMD 100). 
10 Display/non-display is set for the selected component 
(component indicated by 501 in Fig. 5). An operation 
such as the above- described selection operation on the 
operation panel is executed by using the input device 
303. 

15 For example, when the input device 303 includes a 

touch panel and buttons, the operator points at a 
portion on the touch panel with his/her finger. A 
cursor is displayed at the corresponding position in 
the region 506. When the cursor is moved to the 

20 position where the desired selection target is 

displayed, and a button is pressed, the selection 
target is selected. 

A region 507 is used to select the function of 
the 3D pointing device 302. Referring to Fig. 5, "no" 

25 is selected. When, e.g., "laser pointer" is selected 
by using the input device 303, the 3D pointing device 
302 functions as a laser pointer. When the 3D pointing 
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device 302 functions as a laser pointer, for example, a 
linear virtual object having the orientation component 
of the 3D pointing device 302 as a direction vector is 
arranged at the position of the 3D pointing device 302. 
5 With this operation, a laser virtual object which 
extends from the tip of the 3D pointing device 302 
(near the position of the magnetic receiver 203} in the 
direction of the 3D pointing device 302 can be 
arranged . 

10 For example, when the position or orientation of 

the 3D pointing device 302 is manipulated such that the 
laser virtual object crosses a component of an 
observation target virtual object arranged at the 
position and orientation of the 3D pointing device 300, 

15 the component of the observation target virtual object, 
which crosses the laser virtual object, can be 
selected. To help visual recognition of the component 
currently selected by the 3D pointing device 302, 
information about the selected component may be blinked 

20 in the region 506. 

The operation panel window is updated by the 
operation information processing unit 409 in accordance 
with the operation executed in the above-described 
manner, as described above. When an operation of 

25 changing information about the virtual object, and for 
example, the operation of setting display/non-display 
for a selected component is done, the operation 
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information processing unit 409 updates corresponding 
data in the 3D CG scene graph DB 406 in accordance with 
the operation contents . 

The operation panel generation unit 410 receives 
5 operation panel window data for the operation 

information processing unit 409 and outputs the data to 
the operation panel display device 304. The operation 
panel window as shown in Fig. 5 is displayed on display 
screen of the operation panel display device 304. 

10 Fig. 4 is a flowchart showing processing executed 

by the units of the computer 400. 

The position/orientation measuring device 205 
obtains data representing the position and orientation, 
in the sensor coordinate system, of the magnetic 

15 receiver of the 3D pointing device 300 (object 

manipulation device in Fig. 4), as described above 
(4010). By using this data and conversion data , the 
position/orientation information input unit 404 obtains 
the position and orientation, in the virtual space, of 

20 the magnetic receiver of the 3D pointing device 300 by 
known calculation, as described above (4030). Updating 
is done by updating the position and orientation of a 
virtual object associated with the 3D pointing device 
300 in the 3D CG scene graph DB 406. 

25 The position/orientation measuring device 205 

also obtains data representing the position and 
orientation of the magnetic receiver 201 in the sensor 
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coordinate system, as described above (4040). By using 
this data and conversion data, the position/ orientation 
information input unit 404 obtains the position and 
orientation of the magnetic receiver 201 in the virtual 
5 space by known calculation. In addition, the 

position/orientation information input unit 404 obtains 
the positions and orientations of the cameras 102a and 
102b in the virtual space by known calculation by using 
data representing the relationship in position and 

10 orientation between the magnetic receiver 201 and the 
cameras 102a and 102b (4040). The obtained data of 
positions and orientations of the cameras 102a and 102b 
are managed in the 3D CG scene graph DB 406, as 
described above. 

15 In parallel to the above -described processing, 

the image capture units 401R and 40 1L capture the image 
of the physical space from the cameras 102a and 102b, 
respectively (4070), and render the image in a memory 
(not shown) in the computer 400 (4080). 

20 By using data in the 3D CG scene graph DB 406, 

the CG rendering unit 407 lays out the virtual object 
in accordance with its position and orientation 
contained in the data and generates an image of the 
virtual object viewed in accordance with the positions 

25 and orientations of the cameras 102a and 102b (4060). 

The image compositing unit 402R composites the 
image of the virtual space viewed in accordance with 
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the position and orientation of the camera 102a, which 
is generated by the CG rendering unit 407, on the image 
of the physical space captured by the image capture 
unit 401R, and outputs the data of the composited image 
5 to the image generation unit 403R (4090). 

Similarly, the image compositing unit 402L 
composites the image of the virtual space viewed in 
accordance with the position and orientation of the 
camera 102b, which is generated by the CG rendering 
10 unit 407 , on the image of the physical space captured 

by the image capture unit 4011- , and outputs the data of 
the composited image to the image generation unit 40 3L 
(4090) . 

The image generation unit 403R outputs the 
15 composited image (image of mixed reality space) 

received from the image compositing unit 402R to the 
display device 101a. The image generation unit 403L 
outputs the composited image received from the image 
compositing unit 402L to the display device 101b 
20 (4100). 

With this processing, the images of the mixed 
reality space viewed in accordance with the positions 
and orientations of the right and left eyes of the 
operator who wears the HMD 100 on the head sure provided 
25 in front of his/her eyes. 

The operation information processing unit 409 
generates the image of the operation panel comprising 
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the region to hierarchically display information about 
each component of the virtual object and the region to 
manipulate the information about each component by 
using data in the 3D CG scene graph DB 406. When an 
instruction to update the operation panel image is 
input from the input device 303 to select one 
component, as shown in Fig. 5, and display its name in 
a large size, as indicated by 502, the operation 
information processing unit 409 updates the image of 
the panel on the basis of the instruction. 

Hence, the operation information processing unit 
409 always receives an instruction from the input 
device 303 or 3D pointing device 302 (4110). Upon 
receiving an instruction, the operation information 
processing unit 409 updates the panel image in 
accordance with the instruction by, e.g., displaying a 
component name in a large size, as indicated by 501 
(4120 and 4130), and outputs the updated panel image 
data to the operation panel generation unit 410 (4140). 

With this processing, information about each 
component of the virtual object and a result of 
instruction by using the input device 303 are displayed 
on the display screen of the operation panel display 
device 304. 

As described above, according to this embodiment, 
information about each component of a virtual object is 
provided on a relatively small screen, that is, the 
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display screen of the HMD . Hence, the observer can 
easily confirm the information. In addition, the 
operator can manipulate information about each 
component while seeing the screen . 
5 in the above description, the HMD 100 is of video 

see -through type. However, the present invention is 
not limited to this. An HMD of optical see-through 
type may be used. In this case, the cameras 102a and 
102b are not present. Hence, compositing processing 
10 (processing of compositing the image of the physical 
space and the image of the virtual space) is 
unnecessary. 

In the above description, input to the operation 
panel display device 304 is done by using the input 
15 device 303 or 3D pointing device 302. The operation 
panel display device 304 may be a touch panel display 
device. In this case, the input device 303 is omitted. 

In this embodiment, only one 3D pointing device 
(only input device 303) is arranged to manipulate the 
20 position and orientation of the virtual object. 

However, a plurality of 3D pointing devices may be 
. prepared, and one or some of them may be used. 
[Second Embodiment] 

In the first embodiment, the respective units 
25 shown in Fig. 2 as the functional arrangement of the 
computer 400 are formed by hardware. However, some to 
the units shown in Fig. 2 can be implemented by 
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software . 

Fig. 6 is a block diagram showing the basic 
arrangement of a computer 400 which, is implemented by a 
computer such as a PC (Personal Computer) or WS 
5 (Workstation). 

Referring to Fig. 6 , a CPU 601 controls the 
entire computer by using a program or data stored in a 
RAM 602 or ROM 603. The CPU 601 also controls data 
communication with an external device connected through 
10 an I/F 607 (to be described later). The CPU 601 also 

controls the above -described processing for providing a 
mixed reality image to an HMD 100 and processing for 
providing a panel image to an operation panel display 
device 304- 

15 The RAM 602 has an area to temporarily store a 

program or data loaded from an external storage device 
606 and also an area to temporarily store data received 
through the I/F 607 (to be described later). The RAM 
602 also has a work area used by the CPU 601 to execute 
20 various kinds of processing. 

The ROM 603 stores the boot program and setting 
data of the computer. 

An operation unit 604 includes a keyboard and 
mouse and can input various kinds of instructions to 
25 the CPU 601. The operation unit 604 may be used in 
place of the above -described input device 303. 

A display unit 605 includes a CRT or liquid 
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crystal panel and displays a processing result by the 
CPU 601 as an image or characters. 

The external storage device 606 can store a 
program or data to make the OS (Operating System) or 
5 CPU 601 execute each processing which has been 
described in the first embodiment as processing 
executed by the computer 400. Some or all of the 
programs and data are loaded to the RAM 602 under the 
control of the CPU 601. The programs to make the CPU 

10 601 execute each processing described in the first 

embodiment as processing executed by the computer 400 
include programs to make the CPU 601 execute the 
functions of image capture units 401R and 40 1L, 
position/orientation information input unit 404, 

15 operation panel generation unit 410, 

position/orientation calculation unit 405, operation 
information processing unit 409 , image compositing 
units 402R and 402L, CG rendering unit 407, and image 
generation units 403R and 403L in Fig. 3. A 3D CG 

20 scene graph DB 4 06 shown in Fig. 3 is stored as a DB in 
the external storage device 606. 

Cameras 102a and 102b, display devices 101a and 
101b, magnetic receiver 201, position/orientation 
measuring device 205, input device 303, and operation 

25 panel display device 304 are connected to the I/F 607 
so that they can communicate with each other through 
the I/F 607. 
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A bus 608 connects the above -described units. 
[Other Embodiment] 

The object of the present invention can also be 
achieved by supplying a recording medium (or storage 
5 medium) which records software program codes for 
implementing the functions of the above- described 
embodiments to a system or apparatus and causing the 
computer (or CPU or MPU) of the system or apparatus to 
read out and execute the program codes stored in the 

10 recording medium. In this case, the program codes read 
out from the recording medium implement the functions 
of the above -described embodiments by themselves, and 
the recording medium which stores the program codes 
constitutes the present invention. 

15 The functions of the above -described embodiments 

are implemented not only when the readout program codes 
are executed by the computer but also when the 
operating system (OS) running on the computer performs 
part or all of actual processing on the basis of the 

20 instructions of the program codes . 

The functions of the above -described embodiments 
are also implemented when the program codes read out 
from the storage medium are written in the memory of a 
function expansion board inserted into the computer or 

25 a function expansion unit connected to the computer, 

and the CPU of the function expansion board or function 
expansion unit performs part or all of actual 
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processing on the basis of the instructions of the 
program codes . 

When the present invention is applied to the 
recording medium, the recording medium stores program 
5 codes corresponding to the above -described flowchart 
(functional arrangement) . 

As many apparently widely different embodiments 
of the present invention can be made without departing 
from the spirit and scope thereof, it is to be 
10 understood that the invention is not limited to the 

specific embodiments thereof except as defined in the 
claims . 

CLAIM OF PRIORITY 
15 This application claims priority from Japanese 

Patent Application No. 2004-159833 filed on May 28, 
2004, the entire contents of which are hereby 
incorporated by reference herein. 
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